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HOW TO EVALUATE COMPE ITTIVE PRODUCTS 
CEFP 47th ANNUAL CONVENTION, OKLAHOMA CITY 
TUESDAY A. M. GROUP SESSION #5 



FRED WEHLE, JR. - E. F. HAUSERMAN CO. 



Gentlemen, if King Solomon had repeatedly had to make decisions, like 
choosing which of those two women was really the mother of the contested 
baby, he'd know hr> v a facilities planner or his architect must feel. You 
must choose wisely between products superficially similar , but with vast 
differences in future performance for teachers and their students, for 
the community. This is the sort of looking-into-the-future decision 
which would challenge Solomon, or even a Delphic Oracle . Yet with 
great frequency you are doing this, usually with discrimination, and as 
much care as the importance of the decision will allow. But these de- 
cisions are getting tougher all the time. Buildings are becoming more 
complex, and the components becoming larger, as industrialization 
and systems building are becoming more widespread. Now we , a 
couple of sales manager types are supposed to give you some ideas to 
help you evaluate . At most we can suggest a slightly different approach 
or give you an idea you can use. I hope so. This subject , "How to 
Evaluate Competitive Products, " is one you'd think any salesman would 
lcve to talk about. You'd think he'd yearn to seat himself for a v/hile 
in the customer's seat and make wise decisions about this or that major 
purchase for schools. 







od 



In spite of occasional carping about haphazard brand decisions, we sales- 
men in the school market are pretty well impressed with what a good 
job you actually are doing. For you face a difficult problem in keeping up 
with rapid change in both of the two fields you bring together; the fields of 
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education and construction . Most of us feel you face several very difficult 
challenges: first , that of staying current with the rapidly changing needs 
of the teaching professions, is very difficult, with new methods , media 
and programs. Also keeping current with new products and systems in 
the construction industry is very hard indeed, so many changes, whole 
new families of products, new construction methods, and such shifts in 
costs , always upward, but at rates varying from product to product and 
trade to trade. 

Also, you are fettered by the buying practices in the construction industry, 
which are very archaic, structured and oriented not to a value analysis, but 
instead to equating several different items which are assumed to be equal 
and then letting the low general contractor buy from the sub contract bidder 
he can chisel or browbeat the most to get a lower price. As if this lack of 
control weren't enough to try the patience of a saint , you've got some 
weird legal shackles, too. The Federal Government, some stat e and 
even local governments have passed some very restrictive laws, originally 
to protect the taxpayer from the avarice of venal public servants. But now , 
responsible professionals are hampered by these same laws. 

The rules seem to have been set up so that the final brand decision is not 
really a decision at all, but is arranged by fate, " impartially . " There is 
seldom a choice between the final three contenders based on evaluation of 
their merits or their probable relative performance for the use intended. 
Brand "A", "B", and "C" are somehow equated, and the contract awarded 




to the lowest bidder. 



I recently polled some of our salesmen as to how products are typically 



evaluated for school buildings. The consensu s was that architects usually 
do the evaluating, and that because of the press of other matters, and the 
necessity to conserve time ; for most items in a building, this evaluation 
takes place within the architect's office, based on product literature, details, 
performance tests , small samples and perhaps some weight on what has 
already been observed, regarding availability and reliability of installation, 
the sales, and service organizations backing, the proposed products. On this 
basis he specifies accepted products by name , or performance requirements . 

Sometimes the architect visits the field to compare actual installations 
in-use. Usually he is accompanied by the facilities planner. The vie it 
is often promoted by some professional salesman, whom they both trust.. 

But because of the realities of time, geography and slim architectural fees, 
these 'Visits are usually limited to evaluating significant items in the school 
only, perhaps heating, ventilating and air conditioning, lab equipment, 
lighting, demountable or operable walls and some others. 

The evaluation and selection of products seems usually to be a three or four 
step elimination process. The first step is usually during the design develo p- 
ment stage when the architect , his designer or perhaps a facilities planner , 
makes a decision to use a certain generic family or competing products to 
fulfill certain user requirements, selecting some types, eliminating other 
types. 

The second evaluation may take place during the working drawing stage, when 
a narrower spectrum of the type product selected, is drawn-in , as elevations . 



sections and details , more precisely delineating what is wanted. 
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The third step is a further refinement. The specifications , further narrow 
down the field to a relatively few qualified bidders, whose products will 
meet the final criteria. 

Sometimes there is a fourth step, also where instead of seeking the lowest 
priced product to meet minimum performance, the school district seeks 
the best product it can get, under a certain maximum price. Or they seek 
the optimum value for their money, within a maximum amount. 

This requires a value analysis of the products and their expected per- 
formance, for the use intended. This is a more sophisticated purchase 
requiring the exercise of judgment in quantifying the value, and comparing 
values with bid prices . A purchase of this type is usually through the 
process of a bas bid and requested alternates, shown on the general con- 
tractor's bid form, or a purchase directly by the school district. This sub 
contract may then be assigned to the general contractor. 

This subject of evaluation of competitive product should probably be limited 
to the last two evaluations only, as the first two more closely involve 
evaluation of generic types. 

In making selections for a school, there are just too many items required 
to allow much time for visits outside, and on-site evaluation. Increasingly 
architects are relying on performance tests to simulate in-use conditions 
to be anticipated, to get some sort of quantitative evaluation of competing 
products. The test results are submitted and become an important part 
of the evaluation. Partly because of this, the specifications are becoming 
increasingly sophisticated statements of what minimum standards of per - 
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formance are ejected, rather than a description of some preconceived 
idea of how the product should be designed , engineered and manufactured. 

Thus the proprietary br^nd specification with the all inclusive " or equal " 
clause, is giving away to the scientifically prepared performance specifi- 
cation. Such specifications, once written for a project, go a long way to- 
ward making evaluation of products much easier for the architect and the 
facilities planner. For first, a great deal of very specialized technical 
analysis of user needs and environmental forces is needed. 

Painstaking analysis and experimentation is needed to arrive at per- 
formance tests which will simulate and can measure expected in-use pro- 
duct performance. Most school districts can’t afford such research costs. 
But, like the new consumer products which come as " fall-out " from the 
space science program, we are beginning to have available to us fallout 
in the form of the results of some pretty expensive research done for systems 
school building programs like SCSD, SSP and SUCF. This information has 
great potential value for everyone, to help in product evaluation in types of 
construction totally unrelated to systems buildings. 

Look at the specs on large systems projects! The information I am referring 
to is an array of performance tests, which quantitatively measure the 
anticipated performance of building components, subjected to accelerated 
laboratory simulations of the user requirements and the environmental 
forces at work in a school. The Detroit Public Schools Construction 
Systems (CSP) Program, for instance, has a very scholarly , technical 
specification for Interior Space Division , Sub System #4. There are 16 
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different performance tests specified; for finish; relocatability, fire re- 
sistance, impact, sound attenuation, etc. There are also numerous and 
important functional requirements, which are measurable only on a "Yes 
it does it" or "No it doesn't do it" basis, but which are equally important. 

The performance tests submitted by various competitors in response to all 
these paragraphs of the specifications will provide Detroit with the basis for 
an accurate evaluation of the proposed wall systems. 

To Detroit facilities planners and their architects , these ASTM Tests , 
Federal Spec Numbers , etc . mean a very specific level of performance 
to meet criteria , to meet a very specific environmental forces in the new 
buildings.. For instance , there's an impact test where a 60 pound sand- 
bag is dropped three feet onto a 10 foot panel supported at the four corners , 
to measure deflection and set . Why ? Well, you can't really tell from the 
specs. This test is probably designated to reproduce the human environ- 
mental forces present in a busy high school corridor, when a 200 pound 
tackle pushes the full back playfully but hard against the corridor wall. 
Naturally you'd measure the effect of a think like this with an ASTM Test ; 
(E72-61) and you'd limit the permanent set resulting from this to V'. That 
is, you would if you knew for sure why they have that test in the specs, and 
what it is designed to simulate, what sort of abuse it measures. 

What I'm saying is, that these specs are great for the people who wrote 
them, but they don't do milch for the rest of us when it comes to using this 
knowledge on arrother project, because we don’t really know just what they 
are trying to achieve or to simulate with the test. They know though, be- 
cause Detroit did plenty of research on the SCSD buildings and performance 




specs before they wrote theirs. Some of us might think this approach too 
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empirical, because we don't know how they arrived at it, or what the test 
really proves. 

However, at slight extra trouble and expense, the scientific experts who, 
after much research arrived at the test and criteria for acceptance, could 
humanize it and reproduce for the profession and for posterity the problem 
they were solving, as well as their test or solution. Then any architect 
or facilities planner might with impunity use the performance information 
in his own specs, if the problem is similar, and he's satisfied with the 
logic of it. The originator would be sharing knowledge, like a doctor who 
comes up with a new diagnostic breakthrough which he writes about . 

Sharing with others in his profession. All the needed facts are available 
somewhere, because these facts were the original human user requirements 
which were analysed to arrive at the tests, measurement and criteria. All 
we need is to preserve the English language statement of what was intended, 
preferably in the specs themselves. 

A simple example of this user requirement, test and criteria process, is 
the original criteria for operable wall, which we at Hauserman got from 
Educational Facilities Laboratories (EFL) back in 1961. Each requirement 
was humanized, stated in English prose, relative to school and kids. Then 
it was quantified by a test simulating the requirement and stating a minimum 
acceptable performance ! 

For instance, they wanted the wall to provide sound privacy so that a class 
taking a test on one side would not be disturbed by a class reciting on the 
other side. They specified a test, ASTM 119-61 and a sound transmission 




class of 40. 
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They wanted a wall which could be easily operated by any teacher between 
class periods, manually, no electric motors. As the minimum available 
operating force, they selected the equivalent of a small 95 pound female 
school teacher. This force to operate the wall, they then set as 25 pounds 
(after some very human experiments with pretty little 95 pound teachers. ) 

They also specified that the panel surfaces be rigid and flat for accepting 
chalk or tackboards, and that it be simple and maintenance free, etc. 

If the SCSD and Detroit CSP specs had included the user requirements 
which they are testing for, or if they provided some ancilary literature 
for the profession; we could all share in ihe benefits of this basic research. 
But for us to use it, wo must know what they are striving to achieve. 

The University of New York has done precisely this with their interim report, 
’’Interior Finishes Performance Criteria" put out by the State University 
Construction Fund (fondly called SUCF by its friends) This 1968 publi- 
cation gives an array of user requirements for each function referred to, 
and shows use areas to which each level would be appropriate. Then it 
gives test and criteria for each level of requirement. It is however 
limited to interior finishes. It is to have computerized data storage on 
criteria and materials. 

The Toronto SEF Program (Study of Educational Facilities) is probably the 
most exhaustive and extensive study to date. It involves all of the related 
social and technical factors bearing on educational facilities and how they 
can be built more effectively. They published a five volume account of the 
project which does include two on user requirements and others on the per- 
formance specifications which they developed. They worked closely with 
the Canadian Standards Association in selecting appropriate tests and 
developed new standard tests when no suitable tests yet existed. CSA 
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people then checked out these performance tests themselves on the SEF 
Projects' building mock up. 

This research material, the results of millions of dollars and five years 
research in five volumes is available to anyone wishing to purchase it from 
the City of Toronto. Just write to 



The cost is$ U. S. 

If the other agencies managing the systems programs and writing per- 
formance specs, would also provide human user requirements in English 
prose to explain their specs, we'd all be the wiser, and we also could use 
their test information to specify performance, when applicable. As it is, 

I think some current specs are still based on California's SCSD specs 
where the San Andreas Fault and seismic codes were an important con- 
sideration. (How much are we paying for unintended earthquake protection 
in some subsequent school projects in the East?) 

As more school environmental forces and u,^er needs are simulated by per- 
formance tests; let’s share! And let’s humanize specs! 

If we can get these valid measures, we can better evaluate competing 
products. Then let’s use more performance test paragraphs in specs, 
for evaluation purposes. Even where products are called for by brand 
name. 

But evaluation by physical tests usually doesn't give the whole story, and 
won't work well for some products. Here simulation or models may help. 
Mock-ups for testing of function and seeing the appearance, are still used 
for large building projects. 
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When practical, significant brand decisions are being made by facilities 
planners and their architects after visits to see the competing products 
in service in existing schools. To be more objective they usually have 
with them a prepared series of questions to be answered by observation, 
and questioning of teachers and principals. It takes judgment to objectively 
evaluate the level of performance of each competing product in several 
categories. Sometimes these categories are weighted in importance and 
sometimes numerical values are assigned to various levels of performance 
to arrive at an overall picture of relative worth. It helps to organize this 
on a grid. 

As a sales manager, I can tell you that you can get some free help in making 
these evaluations, the competing salesmen who represent the manufacturers 
are experts in product differentiation, but a very biased source of infor- 
mation. How can you use their knowledge yet not accept their bias? 

Though many brands superficially are as like as two peas in a pod, they 
may perform quite differently. The salesmen, if asked, can provide a 
check list for comparing the competing products. Get from each salesman 
a check list well before you are to go. Naturally each check list will stress 
the importance to you of the product characteristics wherein the writer's 
product is strongest. Since you are getting two or more competing check 
lists as a basis, you can come up with your own , list which weighs the 
various factors more equitably, for your own needs, utilizes the sales- 
mens ability to differentiate. If you and your architect go to visit schools 
with various competing salesmen, this composit predone check can help 
you to remember what you observed and make sure you ask the right questions. 
Try to make competing product visits as close together as possible in time, 
so grading will be more consistent. Then enter the facts on a prepared 
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grid to organize your observations and let you assign value points for a 
summary of strong points and weaknesses for a comparative analysis. 

In some states educators are making some post bid brand decisions after 
a value analysis and are opting for the best value rather than just the lowest 
price. Often the lowest bid is not really the best buy, as the GSA recog- 
nizes with its "Life Cycle Cost" evaluation system. With prices spiralling 
upward, there may be some advantage to doing this, to give some flexibility 
to buying after bids are in. 

In summary - Three main points I made were: 

1. Evaluation would be easier, if your spec writer would use more per- 
formance tests to interpret the user requirements and the anticipated 
environmental forces. This would help you evaluate both at the spec writing 
stage to help limit bidding to qualified products only, and after bidding to 
analyze value for the best buy among the brands bid. 

2. The spec writer has reference to specs from big systems jobs but 
needs a Rosetta Stone to decipher them, so let's share, and include user 
needs in the specs, in prose. 

3. Get help free from competing salesmen. Combine their comparison 
check lists to make a value grid with assigned point values for performance 
test results, field observations and reliability. Then compare and evaluate 
brands, perhaps with bid $ amounts shown on the grid also. 

Gentlemen, it was a great pleasure to have this opportunity to talk to you. 

I hope I have been able to offer you something you can use to help you when 
you are again faced with "King Solomon's Choice. " 
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